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ABSTRACT 


This thesis presents and analyses the results of a survey designed to determine how 
Department of Defense financial management education and training programs assess the quality of 
their programs. Quality in the context of this thesis means providing accurate, valid, comprehensive 
and up-to-date information to meet the needs of clients and customers. The thesis explains the need 

within the Department of Defense for financial management education. It documents the methodology 
employed in developing the survey. The thesis discusses the general characteristics of financial 


management education and training programs and summarizes the methods employed to ensure the 
quality of these programs. 
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I. INTRODUCTION 


A. BACKGROUND 

Financial management is one of the most important 
functional tasks performed in the Department of Defense (DoD). 
With an annual budget in the range of $ 290 billion, the 
development of detailed budgets and plans, day to day 
execution, and accounting for the proper expenditure of these 
is a major enterprise. 

A study conducted by the Defense Manpower Data Center 
(DMDC) in December, 1990 indicated that there were 
approximately 119,000 personnel working in or qualified to 
perform financial management related tasks within DoD. This 
is another indication of the magnitude of the effort dedicated 
to financial management. 

The interest of DoD leadership in the financial management 
field is not a recent development and can be traced back prior 
to the mid - 1800's. However, several recent developments 
have increased the awareness within DoD of the importance of 
the financial manager. One of these developments was the 
legislation calling for Chief Financial Officers (CFO’s) to be 
appointed throughout the Federal Government. Another factor 


has been the relative decline in federal budgetary support for 


the military, thus emphasizing the need to carefully budget 











and spend scarce resources. Yet another factor were the 
Defense Management Report Decisions (DMRD) 985 for fiscal 
years 1991 and 1992. These DMRD’s focused renewed attention on 
the financial management community throughout DoD. To briefly 
quote from the first DMRD: 
The financial management community is entering an era of 
new technology, major and rapid change in the domestic and 
international environments, and decreased resources for 
defense. In the future we will have to work smarter, more 
cost effectively, and to respond more quickly than ever 
before. More effective education and training is pivotal 
if the department is to answer this challenge. (Emphasis 
mine) 

In light of the concern over financial management within 
DoD, one of the questions that arose was - how do those 
agencies and departments providing financial management 
education and training courses and programs ensure the quality 
of their offering? This research is a direct result of the 


concern for quality assessment of financial management. 


B. PURPOSE 

The purpose of this research is to identify the quality 
assessment methods used by providers of financial management 
education and training courses and programs offered throughout 
the DoD. The providers of financial management education and 
training courses and programs were asked to complete a 
detailed questionnaire describing their programmatic 
offerings. The survey was used as a vehicle to collect 


detailed information on methods employed by the various 











agencies and departments to ensure the effectiveness of their 
courses and programs. Finally, the study compares the 
various assessment techniques to determine what commonalities 
exist with respect to quality assessment. Similarly, the 
research will examine unique characteristics, if any, of 
quality assessment methods employed in these agencies and 
departments. 

An important by-product, of the research is the development 
of a comprehensive data base of information on quality 
assessment used by all providers of financial management 


education and training in DoD. 


C. RESEARCH QUESTIONS 
The research questions examined in this study are: 
@ What financial management courses and programs are offered 
within DoD and what are the general characteristics of 


these courses and programs? 


@® What methods are employed to ensure the quality of 
financial management courses and programs within DoD? 


D. SCOPE 

The major emphasis of this research is to collect current 
information on quality assessment measures employed by 
providers of financial management and education programs and 
courses within DoD. To accomplish this task, all providers of 
financial management education and training are identified. 


Once this comprehensive list of providers was obtained, these 








agencies and departments were queried as to the methods 
employed to ensure the quality of their programs. 

All military departments and major components of DoD such 
as the Defense Logistics Agency and the Defense Finance and 
Accounting Service are included in the study. In-house as 
well as outside contracted providers also are included in the 
study. Civilian as well as military financial management 
education and training programs are included. Specifically 
excluded from this research are financial management education 


and training programs based on the "correspondence" method. 


E. METHODOLOGY 

The methodology for conducting this research involved four 
distinct steps. Initially, the research concentrated on a 
through review of current literature, instructions, 
directives, reference materials and guidance dealing with 
financial management education and training. 

Secondly, the development of a comprehensive survey 
instrument was undertaken. Survey methodology in the 
literature was reviewed. Prior surveys used by graduate 
Students at the Naval Postgraduate School (NPS) also were 
reviewed for format and answer criteria. Professors and other 
professionals in the field of education assessment were 
queried on appropriate survey questions. A telephone poll was 


conducted with providers of financial management education and 








training to elicit further areas of concern and to develop 
further questions for inclusion in the survey. 

To verify the contents and organization of the 
questionnaire, a pilot test was conducted with two providers 
of financial management education and training. 

Thirdly, the finished survey questionnaire was mailed to 
all identified providers of financial management education and 
training within DoD. 

Finally, the survey responses were analyzed. 

Further discussion of key areas of the iesearch 
methodology employed in this report are presented below. 

1. Literature Review 

The initial sources of data for this thesis were 
developed through a comprehensive literacure search. The 
search focused on three distinct areas. First, the field of 
survey methodology was reviewed to determine the format and 
style of the questionnaire that would be used to gather the 
primary information upon which this thesis is based. This 
rortion of the review included careful attention to the design 
of survey questions, formats of surveys and the method by 
which the data was to be collected, e.g., telephone interviews 
or mailed surveys. 

A detailed examination was also conducted of actual 
surveys used by graduate students at the Naval Postgraduate 


; ; 
School, surveys emp*oyed by educational professionals and 











general purpose surveys which were received by various 
professors at NPS. 

The second area of literature research focused on 
financial management within DoD. Information specifically 
concerned with financial management training course 
availability, course content, and tarceted populations was 
reviewed. Applicable instructions and guidance for financial 
management personnel were also reviewed. Finally, research 
results and recommendations developed in prior studies of 
financial management were reviewed. 

Thirdly, relevant literature on quality assessment and 
assessment techniques was reviewed. 

2. Telephone Interviews 

To help ensure a relevant and comprehensive survey 
questionnaire was prepared, a telephone interview was 
conducted with administrators and professionals employed by 
providers of financial management education and training 
within DoD. Appendix A list those individuals and providers 
contacted in this process. 

3. Survey Questionnaire 

A questionnaire (Appendix B) was distributed to all 
agencies and departments within DoD providing financial 
Management education and training. These agencies and 


departments were identified by using the DoD Training and 








Performance Data Center’s listing of financial management 
education and training providers. 

The questionnaire was designed by a team of Naval 
Postgraduate School faculty and the author to assess how these 
providers ensure the quality of their courses and programs. 
Appendix C lists those agencies that were contacted and 
responded to the survey. Appendix D lists those agencies that 


failed to respond to the survey. 


F. ORGANIZATION OF THE STUDY 

This thesis is organized into five chapters. Chapter I 
provides a general introduction to the area of study. It 
includes a brief background statement, purpose of the 
research, detailed research questions to be examined, scope of 
the study, and methodology utilized in the research effort. 
Chapter II provides background information concerning the 
financial management community, DoD concerns with the 
"effectiveness" or quality of financial management education 
and training, and a brief review of quality assessment 
efforts. Chapter III details the methodology employed in 
developing the survey instrument and collecting the completed 
surveys. Chapter IV analyses the results of the surveys in 
terms of the methods employed to ensure quality. Chapter V 
presents the conclusions reached as a result of the research. 


The chapter also provides answers to the research questions 


examined in the thesis. 








II. FINANCIAL MANAGEMENT 


A. INTRODUCTION 

Management of the federal government’s financial resources 
has become one of the main issues in the debate over the 
growing size of the federal budget and the budget deficit. 
Concern over these issues reaches from the President, 
congressional leadership, and the general public to the 
leadership of our military forces. Efficient and effective 
accounting, budgeting and expenditure of public funds have 
become the watchwords of both the political and military 
leadership within DoD. 

(Ref. 1: pp. 1-8] 

DoD can be compared to the largest corporations in the 
United States. Total employment exceeds 2.6 million 
personnel of which approximately 1.6 million are uniformed 
service personnel with the balance being civilian personnel. 
Budget authority for the 1992 fiscal year budget for DoD is $ 
290.9 billion, larger by far than any corporate budget. 

During the Presidency of Ronald Reagan, the defense budget 
increased considerably. And despite the recent budgetary 
decline (in constant dollars), the need for effective 


financial management has not diminished. In the current era 


of military force structure reductions and concomitant 








budgetary reductions, DoD must ensure that adequate resources 
are available and managed wisely and efficiently to meet the 
defense requirements of the United States. The relative 
decline in federal support for the Defense Department is 
detailed in Table I. Whether in constant dollars or 
discounted for inflation, the decline in funding available to 


the military is clear. [Ref. 2: p. A-6] 





TABLE I. PROJECTED DoD BUDGET OUTLAYS FOR FY 92-97 


1992 1993 1994 1995 1996 1997 Cum % 
Change 
Adjusted $ 266.8 | 247.8 | 237.9 219.3 (27.6) 


Source: The President’s Budget for Fiscal Year 1993 







To meet the ever-growing challenge of decreasing financial 
resources , the financial management community must operate at 
peak efficiency. Education and training programs must be 
designed to develop financial managers capable of dealing with 
the challenges brought about by reduced budgets. 
Additionally, those education and training resources must be 
organized to ensure that the most efficient and effective 
financial management "corp" is available to DoD. The recent 


legislation mandating Chief Financial Officers (CFO’s) for all 


' The adjusted budgetary outlays were deflated by fcur 


percent per year. 











federal agencies and departments is but one of many example of 
the continuing drive to enhance financial management within 
the Federal government. 

To provide a brief guide to the size of personnel devoted 
to financial management within DoD, a study conducted in 1990 
by the Defense Manpower Data Center (DMDC) details the number 
of civilian and service personnel involved in 
financial/resource management. While the aggregate numbers 
presented in Table II below have been questioned by the 
military departments [Ref. 3: p.189] within DoD (e.g., the 
total acknowledged by these components is approximately 20,000 
less than estimated by DMDC), the numbers generally represent 
the total number of personnel engaged DoD-wide in the 
financial or resource management field. Table II details the 
findings of the DMDC study. 

Whether the number of personnel employed in financial 
Management is the 119,000 cited by DMDC or the approximate 
100,000 acknowledged by individual DoD components, the total 
manpower resources devoted to financial/resource management is 


impressive. [Ref. 3: pp. 187-188] 
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TABLE II. FINANCIAL/RESOURCE MANAGEMENT EMPLOYEES OF DoD 





CIVILIAN MILITARY 


Se ee | 46,807, 
jar | a3,o45| 5,647| a,551| 5,376] 26,519 
[wavy | as,523| 9,051 6aa | 2,872| 28, 124 | 
uc | gaa | a,a94| 304] aaa] 3,733. 
foun | 2902] z,e7a| | 5 573 
former | ¢,a2a| sa] | ts 
jroran | 68,520] 32,869| 5,289 | 13,504 | 119,172, 












B. THE INCREASED FOCUS ON FINANCIAL MANAGEMENT 
Financial management or resource management has been 
defined as follows: 


Financial/Resource Management (F/RM) is the 
art and science of acquiring, allocating, and 
controlling the use of resources as expressed 
primarily in monetary terms, but also in terms 
of the physical resources themselves (e.g., 
Manpower and material). F/RM includes the 
functions of Budget Formulation, Development, 
Administration, Review, and Execution; 
Military and Civilian Payroll; Accounting; 
Auditing; Payments; Cost and Economic 
Analysis; Contract Management and Oversight; 
Investment Management; Actuarial Analysis; and 
Management/Program Analysis and Evaluation. 
This definition includes the 
financial/resource management operations 
supporting deployed and battlefield elements 
with the military units established to provide 
such support. [Ref. 3: p. 7] 











As the above definition of financial management 
demonstrates, financial management personnel within the 
military perform a wide diversity of functions. A minimum of 
100,000 personnel within DoD are assigned to financial 
Management activities. 

The need for proper training and education of financial 
Management personnel has been recognized by officials within 
DoD but and by others in the federal government. 

The Comptroller of DoD noted the following: 

The DoD Comptroller is aware of the increased importance 
of education and training for the professional development 
of the members of the financial/resource management (F/RM) 
community. Improvements in the effectiveness and 
efficiency of education and training have become a major 
DoD need. [Ref. 3: p. i] 

The Federal Budget for Fiscal Year 1993 also clearly 
addresses the issue of financial management. As noted in the 
FY 1993 budget, the President requested $ 2.2 billion for 
financial management improvements throughout the Federal 
government, an increase of $ 83 million over funds enacted for 
FY 1992. Both the Fy 1992 and Fy 1993 budgets include 
detailed proposals to "improve financial management". Of 
significance is the following quote from the FY 1992 budget. 
"Good management begins with people. Recruiting and retaining 
a quality workforce is essential to ensuring responsive and 
effective services." [Ref. 4: p. 303] 


DoD financial management personnel must be proficient in 


the basic fundamental financial management concepts, policies 
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and procedures. They must be able to perform their jobs 
competently, effectively and efficiently. Given the 
heightened concern over financial management, DoD must provide 
training and education to financial management personnel that 


ensures the achievement of these goals. 


C. FINANCIAL MANAGEMENT EDUCATION AND TRAINING 

The task of training and educating the 100,000 or more 
personnel involved in financial management is a major 
enterprize. Appendices B and C provide a complete listing of 
the providers of financial management education and training 
within DoD. The training and education provided ranges from 
basic introductory courses in payroll, accounting and 
disbursing for junior enlisted personnel to graduate level 
Master’s programs in financial management for military 
officers and mid-grade civilian employees. 

A total of 29 agencies and departments provide financial 
Management and training. This number does not include the 
correspondence courses offered by the various military 
departments nor does it account for education obtained by 


military and civilian personnel outside of the structured 


programs recognized in this research. The diversity and 
number of courses offered are documented in Table III. (Ref. 
3: p. 127) 
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TABLE ITI. DoD FINANCIAL MANAGEMENT COURSES BY PROVIDER 


Functional area: Arny Navy Air Force DLA Total 


Budget 9 2 aie 13 
Finance 37 23 19 12 91 
Analysis 27 1 19 oe 47 
Resource Mgmt 27 1 -- -- 28 

Totals 100 27 40 12 179 


As Table III demonstrates, the 31 institutions offer 179 
different courses of instruction. While firm data are not 
available, it is estimated that at least 10 percent of the 
financial management workforce, approximately, 10,000 
personnel, attend financial management education and training 


programs and courses each year. [Ref. 3] 


D. RECENT INITIATIVES IN FINANCIAL MANAGEMENT EDUCATION AND 
TRAINING 
The importance of financial management and concomitantly 
the importance of the education and training of financial 
managers has been explicitly recognized in the past several 


years. 
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1. Chief Financial Officer Act of 1990 


One of the more significant events in the area of 
financial management education and training was the passage of 
the Chief Financial Officer Act of 1990. That legislation 
called for the establishment of Chief Financial Officers 
(CFO’s) for the 23 major federal departments. 

In addition to establishing CFO’s for major 
departments within the Federal government, the Act created a 
financial management structure centered in the Office of 
Management and Budget (OMB). Within OMB, a Deputy Director 
for Management, a Comptroller and the Office of Federal 
Financial Management were established to provide a direct link 
with the departmental CFO’s. Secondly, the Act requires 
accountability, i.e., submission of detailed financial plans 
and status reports, from the agencies and from OMB. Third, 
the Act develops a strategy for producing audited financial 
statement for each federal department. These statements are 
designed to professionalize financial management within the 
government and to emulate the practices in corporate America. 

For DoD, the importance of this Act was’ the 
centralization of oversight responsibility for financial 


Management training with the DoD Comptroller. 
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2. Defense Management Report Decision 985. 


Another development in the field of financial 
Management education and training was the initial Defense 
Management Report Decision (DMRD) 985 issued for FY 1991. The 
primary issue that was addressed by DMRD 985 was, "What can be 
done to provide more, and more effective, financial management 
education and training with greater efficiency?" [Ref 5:] 

The main result of DMRD 985 was to charge the DoD 
Comptroller with the responsibility for ensuring that an 
integrated set of high quality, cost effective financial 
Management and education and training programs are provided 
throughout DoD. To accomplish this task, the Defense 
Resources Management Education Center (DRMEC) located at the 
Naval Postgraduate School in Monterey, California was 
assigned the responsibility for advising the DoD Comptroller 
on the progress of implementation of DMRD 985. Additionally, 
the Comptroller was given the authority to designate a 
Director for Financial Management Education and Training to 
manage the education and training of this workforce. Finally, 
the Financial Management Education and Training Working Group 
was established to develop a management plan to delineate the 
roles to be played by the DoD Comptroller and the military 
departments. ([Ref. 5:] 

The Defense Management Report Decision 985 was updated 


for Fiscal Year 1992. (Ref. 6:] The issue stated in this DMRD 
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was, "There is a need to provide more, and more effective, 
financial management education and training with greater 
efficiency." (Compare this definitive statement with the issue 
raised by the original DMRD 985 which asked "What can be done 
to provide more, and more effective, financial management 
education and training with greater efficiency?") The major 
purpose of this revised DMRD was to implement recommendations 
developed as a result of the first DMRD 985. Recommendations 
included: 

1. Establish a structure for the oversight of financial 

management education and training. 


2. Provide career referral services to DoD personnel who are 
not presently provided those services; and 


3. Establish a Resource Management Institute. 


The third recommendation provided that DRMEC would 
become the Defense Resource Management Institute (DRMI). DRMI 
was assigned several functions in the area of financial 
management. Also, DRMI is scheduled to become a Defense 
Support Activity which gives it significant organizational 


status. 


E. THE CONCERN FOR QUALITY ASSESSMENT OF FINANCIAL MANAGEMENT 
EDUCATION AND TRAINING 


Growing interest in financial management and education has - 


been documented in the preceding sections. The Chief 








Financial Officers Act of 1990 and the Defense Management 
Report Decision 985 highlight this concern. One outgrowth of 
this interest in financial management and education was the 
concern by the Dod Comptroller for the "quality" of current 
financial management education and training programs. As 
previously noted, the Comptroller of DoD assumed 
responsibility for the oversight of financial management 
programs. One direct result of this was a memorandum issued 
by the DoD Comptroller on July 3, 1991. The subject of this 
memorandum was "Management Plan for the Review of Financial 
Management Education and Training". [Ref. 7:] While this 
memorandum addressed a broad spectrum of financial management 
issues, it also dealt directly with the issue of quality 
control. Under the heading of Needs Assessment, the area of 
Curriculum Configuration Management and Quality Control was 
discussed. To quote from the Memorandum: 
Financial management course offerings fall generally into 
two categories: core courses that deal in basic concepts 
and principles that are independent of specific 
application ...; and service unique applications that 
train personnel to perform relatively narrow functions or 
apply basic principles in situations peculiar to a 
particular Service. 
In the case of core courses there is a need to ensure that 
the course content covers the subject matter at an 
appropriate level of comprehensiveness and sophistication. 
A second, related function is that of ensuring that what 
is taught is taught well. Whether analyzing new proposals 
or evaluating ongoing programs of instruction, assessment 


of teaching effectiveness are needed. (Emphasis added) 


(Ref. 7:] 
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The specific task of measuring program effectiveness was 
assigned to the Defense Resources Management Education Center 
(DRMEC) . {Ref. 7:] 

This thesis arose as a part of the effort of the DoD 
Comptroller and the Financial Management faculty at the Naval 
Postgraduate School to conduct research on the quality 
assessment methods used in DoD financial management education 


and training programs. 
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III. SURVEY METHODOLOGY 


A. INTRODUCTION 

A principal methodology employed in this thesis is survey 
research. Surveys employed in empirical research typically 
are random, that is, surveys are sent to a randomly selected 
distribution of the targeted population with the expectation 
that only some portion of the surveys will be returned. In 
most cases this approach is satisfactory. However, one of the 
primary goals of this research is to develop a comprehensive 
data base of all providers of financial management education 
and training within DoD and to inventory the various 
techniques that are employed by these providers to assess the 
quality of their programs. Therefore, a census rather than a 
sample of DoD financial management providers was conducted. 

One further note on terminology is required. The terms 
"survey" and "questionnaire" are used in this thesis 
interchangeably. A questionnaire is the tool used in survey 


research to collect information. 


B. THE DATA BASE OF AGENCIES TO BE SURVEYED 

The actual survey that was sent out to all providers of 
financial management education and training programs is 
included in Appendix B. The list of DoD agencies and 


departments responding to the survey is provided in Appendix 
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C. Agencies and departments which failed to respond to the 
survey is provided in Appendix D. 

Prior to developing and mailing the survey to all 
providers of financial management education and training, a 
comprehensive list of these providers needed to be verified. 
The principal sources for determining the agencies and 
departments to be surveyed were: 

(1) Formal Schools Directory, Third Edition [Ref. 8:] 
dated January, 1992. This directory was prepared by the DoD 
Training and Performance Data Center (TPDC) which is located 
in Orlando, Florida. The directory lists all schools and 
training locations operated by the Military Services and DoD 
agencies, including those identified as financial management. 
Five hundred eighty nine training organizations are identified 
in this directory. 

(2) Financial Management Data System [Ref. 9:] also 
developed by TPDC. The Financial Management Data System is an 
automated data collection system which is designed to 
facilitate the collection of descriptive and resource data for 
DoD financial management training and education courses. TPDC 
provided this researcher with the latest edition of the data 
base which was updated through January, 1992. 

The financial management data base was cross-referenced to 
the Formal Schools Directory [Ref. 8:] to ensure that the 
latest available data on providers of financial management 


education and training was utilized. 
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A final check was performed to ensure that all providers 
of financial management education and training were included. 
This included correspondence and communication with experts 
both at the Naval Postgraduate School and with field personnel 
in the various military departments. The final product or 


listing of providers was then prepared. 


C. DEVELOPMENT OF THE SURVEY INSTRUMENT 
1. Research on Survey Methodology 

The development of the survey used to gather data for 
this research began with an extensive review of the literature 
on survey design. [See references 10 through 21] One of the 
first issues that needed to be addressed was what form would 
the survey questions take, e. g., open or closed format, 
scaler, filtered. "Open" questions are designed to allow the 
widest latitude to survey recipients in responding to the 
survey question. An example of an open question would be: In 
your opinion, what are the most important methods you employ 
to ensure the quality of your program? Closed questions, on 
the other hand, present a listing of options or preconceived 
responses for the survey respondence to choose from. The 
scaler type of question asks the recipient to assign a "score" 
to a particular, question. This score normally ranges from 5 
to 1 if numerically weighted or from frequently to seldom in 


verbal terms. Filtered or filtering questions involve the use 
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of "no opinion" type responses or simple "YES/NO" responses. 
(Ref. 11: pp. 41-56] 

Since the full spectrum of assessment techniques 
employed by DoD financial management education and training 
programs was not known, the basic methodological approach 
employed in the development of the questionnaire was the 
"open" response format. This format allows the recipient to 
reply to the questions in a manner best suited to the 
institutional setting of the respondent. (Ref. 11: p. 54] 
Additionally, filtering questions were developed to facilitate 
the response to questions that did not apply to certain 
recipients but did apply in general. Finally, general purpose 
or information questions were included in the question data 
base. 

A second methodological issue that was addressed was 
who should be contacted in the survey process. Mr. Peter 
Ewell, Senior Associate with the National Center for Higher 
Education Management Systems (NCHEMS) noted that quality 
assessment of education could be conducted in two principal 
ways. These methods are: 

(1) collection from the agencies and departments 
themselves or an internal perspective, and 

(2) Collection of information from the clients served by 
the agencies and departments. The clients are either the 
students themselves or the sponsoring agencies which 


subsequently received the students. This method of data 
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gathering is referred to as external. [Ref. 22] Due to the 
limited time frame and the prospect of uncertain response from 
the external sources, the internal method was selected. 

The third methodological issue addressed was the data 
collection method to be employed in the survey. The various 
methods of data collection are presented in Table IV below. 
The table lists the methods, a brief description of the method 
and summarizes the weaknesses and strengths of each method. 
The information derived in Table IV was derived from 
Measurement Errors in Surveys by Paul P. Biemer. 

(Ref. 11: pp. 237-250] 

Given the time constraints as well as the perceived 
length of the questionnaire on financial management quality 
assessment techniques, a combination of telephone and mail 
survey techniques were selected. In addition to the 
probability of gaining higher reliability of the responses, 
this method allowed the survey respondent to include 
additionally ma erials to the basic survey document. The 
problem of non-response, which is normally a factor in a mail 
survey, was discounted due to the support for the survey by 
the Comptroller of DoD. This is discussed later in this 


section. 
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Methods of 
Data 
Collection 


Questionnaire 


Observation 


TABLE IV. 





Description 


Survey mailed 
to respondent 


Agent observes 
activity to be 
measured. 


Agent collects 
survey data 
via phone. 


Survey 
administered 
in presence of 
respondent. 


DATA COLLECTION METHODS 


Advantages 


Data from 
large number 
of people. 
Standard for 
measuring 
change. 
Quick. 


Rich 
behavioral 
data. 


Similar to 
questionnaire. 


Provides best 
data. 





Disadvantages 


Measure 
attitudes not 
behavior. 
Requires 
honesty on 
part of 
respondent. 


Time 
consuming. 
Subjective. 


Consistency. 
Time 
consuming. 


Time 
consuming. 
Interview 
bias. 


Source: Paul P. Biemer, Measurement Errors in Surveys, 1991. 


2. Initial Development of the Survey Questions 


Once the decision had been made by DoD Comptroller 


staff to perform a quality assessment of financial management 


education and training courses and programs, 


development of the survey instrument was undertaken. 


phase of development, two steps were followed. 


the initial 


In this 


The first step 








was to review research literature on survey methodology. 
(Detailed in the previous section). 

The second step was to contact professionals in the 
field of educational assessment to provide additional guidance 
on the survey design. 

One of the more significant contributions at this 
stage was provided by Peter T. Ewell, Senior Associate at the 
National Center for Higher Education Management Systems 
located in Boulder, Colorado. In his response to a request 
for assistance in developing a comprehensive survey, Mr. Ewell 


recommended the following procedures be assessed: 


@ The nature of the program with respect to target audience 
(e.g., civilian/military, rank, geographical region) and 
its intensity/duration (e.g., one day, two-week on site, 
one year part-time, correspondence course). 


@® The learning objectives of the program, framed if possible 
in terms of "expected outcomes" of instruction i.e., 
particular elements of knowledge or skill that a student 
will exhibit at the programs conclusion. 


@® The particular methods or modes of instruction used in 
delivering the program (e.g., hands-on training, 
simulation, classroom work, independent study, etc.), and 
in particular the ways in which students’ competency is 
tested at its conclusion (e.g., paper-and-pencil exams, 
problems, rated demonstrations of hands-on performance, 
etc.). 


@® How the effectiveness of the program in attaining its 
training objectives has been evaluated in the past; if 
possible, units should be requested to attach copies of 
any studies or data on effectiveness that they routinely 
collect to monitor and improve their own performance. 
(Ref. 22] 
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Concurrent with seeking outside guidance, a team of 


Naval Postgraduate Financial Management professors provided a 
series of questions for inclusion in the survey. These were 
in addition to the questions developed as a result of the 
literature search and contact with educational professionals 


outside DoD. 


3. Field Validation of the Survey Questions 

Once the initial set of survey questions had been 
compiled, the "validity" and "relevance" of these questions 
needed to be tested. To accomplish this testing, selected 
agencies and departments offering financial management 
education and training programs were contacted through a 
telephone survey. The methodology employed in this telephone 
survey was "blind" response, i.e., the respondent was asked 
general questions on quality assessment and then asked what 
questions they would consider relevant in developing a survey 
instrument for their facility. Ten of the 31 institutions 
previously identified as providers of financial management 
education and training were contacted in this process. The 
providers were selected on the basis of size, service brz ach 
and referral from previously contacted providers. The various 
questions, suggestions and pertinent information collected 
during the telephone survey were added to the data base of 


questions prepared prior to the telephone survey. Appendix A 
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lists those agencies and departments contacted during this 
phase of survey preparation. 
4. The Initial Survey Instrument 

Once the agencies and departments were selected for 
survey and the survey questions to be asked had been collected 
and developed, the next phase was to organize the survey 
questions into logical units. Specifically, the questions 
that had been obtained through the research noted previously 
had to be organized into a cohesive survey document. The 
organization of the survey was viewed as essential for 
complete and successful data collection on the quality 
measures employed by financial management education and 
training providers. 

As Paul Biemer notes in Measurement Errors in Surveys 
"(Survey] Questions are not asked in isolation but are grouped 
together in a questionnaire." [Ref. 11: p. 38] Further, "Once 
the researcher decides [what questions to ask], he needs to 
determine the order in which they are presented to the 
respondents. Survey literature indicates that presentation 
order strongly influences the... results." (Ref. 11: p. 51] 

One typical organizational model that may be employed 
is the "process" or "throughput" model. This model 
essentially has three components - inputs, outputs, and the 


processing o: inputs to outputs. 
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Various studies on evaluation [Refs. 10, 12, 14, 17, 

18, 19, 20, 21) have noted variations to the simplified 
process model described above. For example, Suchman 
identified five types of evaluation that an agency can employ 
to measure its performance. He defined the types as follows: 
de BESTE « Effort is equivalent to input. Effort 


evaluation is measuring inputs as an indicator of meeting 
the goals and objectives of an organization. 


2. Effectiveness. Effectiveness is the measuring of the 
outputs of an organization. 

3. Adequacy. Adequacy measures outputs in a larger 
institutional setting. It measures outputs to needs, 


i.e., why are the services being provided? 


4. Efficiency. Efficiency attempts to relate outputs to 
levels of inputs. 


5. Process. Process focuses on the process by which inputs 


are transformed into outputs. 
[Ref. 16: pp. 64-65] 


Stufflebeam uses a similar dichotomy to describe the 
evaluation process. He divides evaluation into context, 
input, process, and product categories. [Ref. 19: p. 64] 
Stufflebeam’s categories differ slightly from Suchman’s but 
essentially view the organization in a similar vein. The main 
difference is that the timing of evaluation is key to 
Stufflebeam’s model whereas Suchman is more concerned with 
organization processes. 

Based on the research conducted on organizational 


models, the process model for organizing the survey instrument 
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was adopted due to its inherent simplicity. Questions were 
segmented into the following categories (which roughly 
correspond to various phases in organizations). The first 
three categories correspond to inputs, the fourth to process 
and the last to output. The general categories and principle 


questions or information requested is listed below: 


Ls neral organizational and program inf ion includin 
respondent, point of contact, target audience, number of 
financial management courses and programs offered, length 
of the program, numbers of times offered, annual 
enrollment and average class size among others. 


2. Progr xr lrements an velopment: How is the need 
determined for new courses? How is the need tc update 
or modify courses determined? Is course development 
guided by higher authority? and Does the agency or 
department have staff dedicated to program development? 


3. Instruction: What policies are used to hire 
faculty/instructors? What methods are use to evaluate 
instruction and instructors? 


4. Ongoing evaluation: What methods are used to determine 
the need for the current program? What methods are used 
to determine student competency? Are formal methods used 
for student course evaluation? and What percentage of 
students successfully complete the program? 


5. Program assessment: Is the performance of graduates 
tracked? What formal methods are employed to solicit 
feedback from former students and clients of those 
students? 

Once the questions were organized as noted above, the 


complete survey questionnaire was ready for initial pilot 


testing. 
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5. Pilot Testing 

Pilot testing consisted of having the survey document 
reviewed by field personnel to determine if the survey made 
sense, was comprehensive and addressed the primary areas of 
financial management quality assessment. 

The survey was given to CDR Glenn Eberling who taught 
the Navy Practical Controllership Course at the Naval 
Postgraduate School in Monterey, California and to Mr. Wade 
Cliendienst who was the Chief of Training and Professional 
Development for the US Army Audit Agency located in 
Alexandria, Virginia. 

Both recipients reviewed the survey and offered 
several additions to the proposed survey instrument. These 
additions focused on the methods of quality assessment and the 
organization of the survey. Both pilot recipients also 
suggested clarification in the wording of the questions and of 
the explanatory paragraphs included in the survey. Neither 
Mr. Cliendienst nor CDR Eberling are responsible for the 
ultimate content of the survey. Their contribution to the 
survey was important and therefore merits recognition. 

6. Sponsorship by Comptroller of the Department of 

Defense 

A final step in developing the survey of financial 


Management providers was to obtain the support of the 


Comptroller of DoD. As noted, this thesis was the result of 











the desire by the DoD Comptroller and his staff to assess the 
quality of financial management education and training 
programs. Naval Postgraduate School Financial Management 
Faculty assisted in this process through direct liaison with 
the DoD Comptroller. 

A draft cf the survey was reviewed by DoD Comptroller 
staff and a cover letter from DoD Comptroller Sean 0’ Keefe was 
provided to introduce the survey. Sponsorship of the survey 
by the Comptroller was essential to ensure support from the 


agencies and departments surveyed. The cover letter from the 


DoD Comptroller is included at the end of Appendix B. 








Iv. ANALYSIS OF SURVEY RESPONSES 


A. INTRODUCTION 

This chapter summarizes the findings obtained from the 
survey responses provided by financial management education 
and training providers. Appendix E provides a comprehensive 


review of all survey responses from which this chapter was 


developed. 
Of the 31 surveys sent to providers of financial 
Management education and training, 21 were returned’. This 


represerts a response rate of 68 percent. Surveys returned by 
the Center for Army Leadership and the Army’s Judge Advocate 
General’s School indicated that neither agency offered 
financial management education and training. 

The response rate based on a revised total of 29 providers 
(the original thirty-one less the two mentioned above) with 19 


responses is 67 percent’. 


* The 21 responses were received prior to April 30,1992, 


the cut-off date for inclusion in the research discussed in 
this chapter. | 

3 An effort was made to obtain a 100 percent response 
rate to the survey. Three non-responding agencies were 
specifically identified for intensive follow-up due to either 
their size or number of course offerings. They were the DLA 
Finance Training Section, the Defense Systems Management 
College and the Army Audit Agency. As a result of these 
efforts, surveys from the Defense Systems Management College 
and DLA Finance Training Section were received after the 
primary research on the thesis had been completed. The 
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Another measurement of the response is to calculate the 


number of financial management programs managed by agencies 
and departments who responded to the survey. By this 
reckoning, the total number of financial management programs 
offered totals 181. Respondents to the survey manage 134 of 
the 181 programs. The response rate based on this measure is 
79 percent’. 

Tables V(A) and V(B) detail the survey questions asked of 
respondents and the actual number of responses provided to 
each question. The variability in responses to the questions 
is based on (1) the non-applicability of certain question to 
the various respondents and (2) the failure of certain 
respondents to answer specific questions. The variability in 


response rate effects the discussion which follows. 


responses from these two agencies conformed to the results 
obtained from the nineteen survey respondents noted in this 
section. Also, Don Cress of the Army Audit Agency reported 
that he intended to complete and return the survey. A revised 
response rate based on 22 responses is 76 percent. The other 
non-responding agencies were contacted if the point of contact 
was known, but no other follow-up was conducted. 

* Including DLA, Defense Systems Management College and 
the Army Audit Agency, the response rate is 87 percent; 158 of 
181 programs. 
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TABLE V(A): SURVEY QUESTIONS/NUMBER OF RESPONSES 


SURVEY QUESTION 


Le TARGET AUDIENCE 

2. PERCENTAGE OF CLASS ATTENDEES 
3. GRADES/RANKS OF ATTENDEES 

4. NUMBER OF COURSES 

5 LENGTH 


HOW OFTEN IS GUIDANCE RECEIVED 
STAFF DEDICATED TO PROGRAM 





DEVELOPMENT 


1 


8. 


PERCENTAGE OF MIL./CIV. INSTRUCTORS 


23. OTHER METHODS TO EVALUATE 
INSTRUCTION 

24. OTHER METHODS TO EVALUATE 
INSTRUCTORS 
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B. SURVEY RESPONSES 
1. General Characteristics of Financial Management 

Education and Training Courses and Programs 

Of the 21 responses received in response to the 
survey, 19 agencies provided detailed responses and two 
agencies replied that they did not conduct financial 
Management education and training. 

The survey questionnaire was designed tc gather basic 
data on the characteristics of financial management education 
and training providers. Questions one through ten provided 
the respondents with the opportunity to describe their courses 
or programs in terms of target audience; number and length of 
those courses and programs; enrollment; average class size;and 
the level of instruction ( e. g., degree granting). Several 
other questions also provided general program information on 
these providers. For example, question 32 asked whether the 
financial management program was supported by a library 
containing extensive DoD financial management’ related 
materials? The following material details the general 
characteristics of financial management education and training 
providers responding to the survey questionnaire. Table VI 
summarizes the general characteristics of financial management 
education and training providers. 

The respondents represented a wide spectrum of 


institutions providing financial management education and 
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training. In terms of military sponsorship, five respondents 
were components within or sponsored by the Department of the 
Army, three within the Department of the Air Force, and eleven 
by the Department of the Navy (including one response from the 


Marine Corps). 


TABLE VI: UMMARY OF GENERAL TERISTIC 


QF FINANCIAL MANAGEMENT EDUCATION AND TRAINING PROVIDERS 


GENERAL CHARACTERISTICS NO. OF RESPONSES 


MILITARY SPONSORSHIP: 

ARMY 5 
3 
pipe 
1 
9 
9 
3 
1 
5 —— 
4 
4 



































PROFESSIONAL 
FULL SPECTRUM 


AVERAGE ENROLLMENT: 
LESS THAN FIFTY 
FIFTY TO ONE HUNDRED 
ONE HUNDRED ONE TO TWO HUNDRED 
MORE THAN TWO HUNDRED 


NUMBER OF COURSES OFFERED: 
ONE 
TWO TO FIVE 
SIX TO TEN 
MORE THAN TEN 


TARGET AUDIENCE: 
MILITARY 

CIVILIAN 

BOTH 


FACULTY COMPOSITION: 
MILITARY 

CIVILIAN 

BOTH 


AIR FORCE 
2 
3 
2 
10 


NAVY 
LEVEL OF PROGRAM: 
ENTRY LEVEL 
| + | 
| £ : 
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In addition to representing all of the military 
services, the respondents spanned the educational spectrum 
from those providing basic military entry level training to 
those granting Master degrees. Nine of the nineteen 


respondents were classified as providing professional level 


education (e. g., serving higher level civilian and military 
personnel or providing an advanced educational degree -- See 
question 3., Appendix E.) . Of these nine, five offered 


graduate level programs leading to the award of a master’s 
degree. 

In terms of the size of the institutions responding to 
the survey, average enrollment in programs or courses ranged 
from a low of 36 to a high of 1,670. The total annual student 
population as reported by the respondents was 7,324. Another 
measure of size is the number of courses offered. The range 
of courses offered was from one to 36, with the average (mean 
and median) being eight. 

The target audience as reported by the survey 
respondents, not surprisingly, was both military and civilian 
employees of DoD. Only one agency reported its audience as 
civilian only, and three responded military only. Fifteen 
agencies served both. 

In terms of faculty composition, e. g., military, 
civilian or a combination of both, four agencies used civilian 
instructors, four military and eleven had a mixture of 


military and civilian. 
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The wide diversity in program offerings, size, 
instructors and military sponsor of the respondents ensured 
that the survey responses represented the entire spectrum of 
financial management education and training providers. 

2. Quality Assurance Methods Employed By Providers of 


Pinancial Management Education and Training 


The survey .questionnaire was designed to elicit 
information on the quality assessment methods that are used by 
financial management education and training providers. Since 
the survey was a self-assessment of these methods, the 
responses provided represent the views or opinions of the 
providers on quality assurance. 

No attempt was made to force the responses into a 
preconceived model of quality assurance. Since the survey is 
descriptive, the responses by providers of financial 
management education and training represent their 
interpretation of quality assurance and the importance they 
attach to the methods used to assure quality in their courses 
and programs. 

The analytical framework employed in the following 
discussion begins with a detailed analysis of Question 47. 
Question 47 of the survey asks -- What are the most important 
methods you employ to ensure the quality of your program? 


(Emphasis added). The question provided respondents the 
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opportunity to summarize the methods they used to assure 
quality. The responses to this question provide the basis for 
discussing the quality assurance methods employed by financial 
management education and training providers. 

The following listing summarizes those methods cited 


by survey respondents to ensure the quality of their programs: 


@® Student feedback (11 responses). 

@® Sponsor guidance/feedback/support (6 responses). 
@ Hiring/ensuring quality faculty (6 responses). 

@ Feedback/contact with clients (9 responses). 


@ Tracking the post-graduation performance of students 
(5S responses). 


@® Monitoring of faculty performance (3 responses). 


@ Keeping abreast of changes in financial management 
(4 responses). 


@® Tracking student performance in courses and/or programs 
(2 responses). 

The methods noted by financial management education 
and training providers in their responses to this survey 
question (Question 47) are supported by the responses made to 
other questions within the survey questionnaire. Each of the 
areas cited above are separately discussed below with 
references to other survey questions as appropriate. 
Responses to Question 47 are not included inasmuch as that 


question forms the basis for further analysis. [Complete and 
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detailed responses to the survey questions are provided in 
Appendix E, and are not specifically quoted below] 
a. Student Feedback 
The use of student feedback, specifically end-of- 
course critique forms, is a response that appears throughout 
the surveys returned by financial management education and 
training providers. In addition to Question 41 which asks -- 
Do you have a formal method of student course evaluation? to 
which all nineteen respondents affirmatively replied, the 
following questions and the number of respondents that listed 
student feedback as a prime determinant are as follows: 
@® Question 11: How do you determine the need for new 
courses or instruction? (6 responses) 
@ Question 12: How do you determine the need to update or 
modify the instruction you currently offer? (10 
responses) 


® Question 23: Are other methods used to evaluate classroom 
instruction? [Note: Question 23 asks whether respondents 


use classroom observation to evaluate instruction] (15 
responses) 
@ Question 24: Are other methods used to evaluate 


instructors? (7 responses) 


@® Question 38: What methods are used to determine the need 
for your current program? [Follow-on to questions 11 and 
12} (3 responses) 


@® Question 39: What methods are used to determine the 


validity, accuracy, relevance and timeliness of your 
current program? (7 responses) 
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b. Sponsor Guidance 

Sponsor guidance is another common method of 
quality assurance employed by respondents to the survey. 
Question 13 (Do you have one or more sponsors or clients that 
directly influence your program of instruction?) directly 
measures this question. Eighteen of the nineteen respondents 
acknowledged sponsor guidance. Question 14 (Is course 
development guided or directed by higher authority?) also 
addresses this area. Thirteen of nineteen respondents 
responded "yes" to the question. 

Additionally, the following survey questions 
applied to the sponsor guidance: {[Again, the number of 
positive responses is appended to the end of each question] 

@® Question 11: How do you determine need for new courses or 
instruction? (9 responses) 


@® Question 12: How do you determine the need to update or 
modify the instruction you currently offer? (8 responses) 


@ Question 27: Is your institution reviewed by other 
agencies [other than accreditation]? (7 responses) 


@ Question 38: What methods are used to determine the need 
for your current program? (6 responses) 


@® Question 39: What methods are used to determine the 
validity, accuracy, relevance and timeliness of your 
program? (8 responses) 

c. Hiring Quality Faculty 
Hiring of a quality faculty was the third quality 


assurance method cited by survey respondents. In this regard, 
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the issue of faculty "quality" or competence was addressed 
directly by the survey in questions 19 through 25. (See 
Appendices A and E) Question 21 is the most specific. It 
asked - Are professional qualification standards required for 
faculty/instructors? Seventeen of 18 respondents stated that 
they employed this method. 

Also relevant for the discussion of quality 
faculty were questions 19 and 20. The questions asked - What 
policies and criteria are used to hire military (Question 19) 


or civilian (Question 20) faculty/instructors for your 


program. Survey responses to both questions focused on 
experience, education and other factors that prepare 
individuals for the teaching profession. In fact all 


respondents noted that they used selected criteria to ensure 
the hiring of qualified individuals. 

Question 22 through 25 further addressed the issue 
of instructor or instruction observation to promote quality 
faculty. However, the issue of faculty on-the-job 
performance, while relevant to a quality faculty is more 
appropriately discussed in subsection (f) Faculty Performance 
below. 

d. Feedback from Clients 

Client feedback was the fourth method cited by 

survey respondents to ensure quality. Question 13 addressed 


this method by asking -- Do you have one or more sponsors or 
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clients that directly influence your program of instruction? 
As the question itself implies, the distinction made between 
clients and sponsors was not always made clear in the 
responses provided by respondents. Only one respondent 
specifically mentioned a client that influenced their program. 
Nevertheless, other survey question responses noted the 
importance that financial management education and training 
providers gave to client feedback. 
The following questions directly applied: 
® Question 11: How do you determine the need for new 
courses or instruction? (9 responses) 


@® Question 12: How do you determine the need to update or 
modify the instruction you currently offer? (3 responses) 


@ Question 38: What methods are used to determine the need 
for your current program? (2 responses) 


@® Question 39: What methods are used to determine the 
validity, accuracy, relevance and timeliness of your 
current program? (3 responses) 

Additionally, client feedback is also obtained 
through the use of post-graduation surveys to the extent that 
graduates and their supervisors represent the clients of the 
courses and programs offered. This area is more fully 
examined below. 

e. Post-graduation Performance of Students 

The post-graduation performance of financial 

management education and training students was the fifth 


method of quality assurance indicated by survey respondents. 
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This method was specifically addressed in the survey in 
Question 45. The question stated -- Do you track the 
performance of your graduates? Seven of the nineteen 
respondents did utilize such tracking. Of these seven, six 
used post-graduation surveys for this purpose. 

f£. Faculty Performance 

A sixth method of quality assurance reported was 
faculty performance. This method is similar to the hiring of 
quality faculty previously discussed in subsection (c). 
However, the monitoring of faculty performance is an on-going 
process that, in principle, verifies the hiring policies and 
procedures of financial management providers. 

Questions 22 through 25 addressed the issue of 
faculty performance. These questions with the number of 
affirmative responses are as follows: 

@® Question 22: Do you use classroom observation to evaluate 
instruction? (17 responses) 


@® Question 23: Are other methods used to evaluate classroom 
instruction? (15 responses) 


@® Question 24: Are other methods used to evaluate 
instructors? (12 responses) 


@® Question 25: Do you employ productivity measures to 
evaluate instructors? (3 responses) 

One of the prime measurements of faculty 

performance was the use of student critiques (end-of-course 


evaluations) and student feedback. For example, all fifteen 


46 








of the responses to question 23 cited the use of student 
critiques as the method used to evaluate classroom 
instruction. Similarly, eight respondents to question 24 
noted student critiques as the primary method employed to 
Measure instructor performance. The method of student 
feedback as a quality assurance tool is discussed in detail in 
subsection (a) above. 
g. Keeping Abreast of Changes 
Keeping abreast of changes in financial management 
was the sixth quality assurance method cited by survey 
respondents. No specific survey question explicitly dealt 
with this method. However, the need for financial management 
providers to maintain currency of their courses and programs 
to ensure timeliness and relevance is obvious. The need to 
keep abreast of changes in financial management was noted in 
the following survey questions: 
@® Question 12: How do you determine the need to update or 
modify the instruction you currently offer? (5 responses) 
@® Question 39: What methods are used to determine the 
validity, accuracy, relevance and timeliness of your 
current program? (2 responses) 
h. Tracking Student Performance 
The final method reported to ensure quality was 
the tracking or monitoring of student performance while 


attending the financial manac2ment institution. 
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The capability of an institution to measure the 
performance of its students informs that institution on the 
success or failure of its instruction. The survey addresses 
the issue of student performance or competency in questions 
40, 43 and 44. 

Question 40 asked -- What methods are employed to 
determine student competence during and upon program 
completion (e. g., passing standardized tests, written course 
work, observation of performance on the job)? Question 43 
deals with the requirement for students to pass national 
examinations. Only one respondent required this. Finally, 
Question 45 ask respondents what percentage of students 
entering the program successfully complete it? The average 
successful completion rate was in excess of 95 percent. 

Additionally, while this method is directed toward 
in- house success, post-graduation performance also informs 
the institution on the quality of their program. This issue 


is addressed in subsection (e). 


C. SUMMARY 

The preceding sections discussed the responses to the 
survey of financial management education and training 
providers in terms of (1) the number of responses; (2) the 
general characteristics of financial management providers; and 
(3) the methods that financial management providers used to 


ensure quality. Table VII summarizes the methods used. The 
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number of responses cited in the table represent those 
respondents that acknowledged the use of the quality method. 
The percentage column is calculated on the basis of the number 
of responses divided by the total number of responses received 


(i. e@., nineteen). 


TABLE VIL. SUMMARY OF QUALITY ASSURANCE METHODS 


[(b) Sponsor Guidance | | 
|(c) Hiring Quality Faculty | 6 =| at 
|(a) Feedback from clients | 9 | 7 
[(e) Post-graduation Performance | 5 | 23.3 
[(f) Faculty Performance | 3 | 58 
[(g) Keeping Abreast of Changes | 4 | 2.1 
[(h) Tracking student Performance | 2 | 10-5 
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Vv. CONCLUSIONS 


A. INTRODUCTION 

This thesis presented and analyzed the results of a survey 
designed to determine how DoD financial management education 
and training programs assess the quality of their programs. 
Quality in the context of this thesis means providing 
accurate, valid, comprehensive and up-to-date information to 
meet the needs of clients and customers. 

The thesis described the need within DoD for financial 
Management education. It documented the methodology employed 
in developing the survey. The thesis discussed the general 
characteristics of financial management education and training 
programs and summarizes the methods employed to ensure the 


quality of these programs. 


B. ANSWERS TO RESEARCH QUESTIONS 

The two research question which guided this thesis were: 

@® What financial management courses and programs are offered 
within DoD and what are the general characteristics of 
these courses and programs? 

@ What methods are employed to ensure the quality of 
financial management courses and programs within DoD? 
Chapter IV presented the findings of the research. The 


first research question addressed the general characteristics 
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of financial management education and training programs within 
DoD. Table VI in Chapter IV summarizes these characteristics. 

The respondents represented a wide spectrum of 
institutions providing financial management education and 
training. In terms of military sponsorship, five respondents 
were components within or sponsored by the Department of the 
Army, three within the Department of the Air Force, and eleven 
by the Department of the Navy. 

The respondents spanned the educational spectrum from 
those granting Master degrees to those providing basic 
military entry level training. In terms of the size of the 
institutions responding to the survey, average enrollment in 
programs or courses ranged from a low of 36 to a high of 
1,670. The range of courses offered was from one to 36, with 
the average (mean and median) being eight. 

The target audience as reported by the survey respondents 
was both military and civilian employees of DoD. Only one 
agency reported its audience as civilian only, and three 
responded military only. The remainder served both. | 

In terms of faculty composition,e. g., military, civilian 
or a combination of both, four agencies used civilian 
instructors, four military and eight had a mixture of military 
and civilian. 

The second research question addressed the methods of 
quality assurance used by financial management providers. The 


responses by survey respondents to the question of methods 
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used to ensure quality are listed below: (Each of these 
methods was extensively reviewed and discussed in Chapter IV, 
pages 39 to 49, and summarized in TABLE VII) 

(a) Student Feedback 

(b) Sponsor Guidance 

(c) Hiring Quality Faculty 

(d) Feedback from Clients 

(e) Post-graduation Performance 

(£) Faculty Performance 

(g) Keeping Abreast of Changes 

(h) Tracking Student Performance 

As noted in Chapter IV, student feedback and feedback fram 
clients were the two quality assurance methods most commonly 
cited by survey respondents. The survey results, however, 
clearly point out the wide divergence of quality methods used 
by financial management providers. 

In addition to the methods cited by survey respondents to 
ensure quality, there are other quality measures not mentioned 
in the survey responses. The following two methods are 
offered as examples of other quality assurance tools that 
could be employed by financial management education and 
training providers: [This list is not intended to be all 
encompassing; other methods are certainly possible. The 


discussion follows the format used in Chapter IV, Section 2] 
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(1) Accreditation 


The accreditation process provides an independent 
check on the performance of participating agencies and 
departments. Accreditation organizations typically review 
courses and programs for content, instructor competence, the 
provision of ancillary services such as libraries, etc. 

Question 26 asked - Is your institution accredited? 
Nine of the nineteen respondents replied affirmatively. The 
accrediting organizations included the North Central 
Association of Colleges and Universities, Commission on 
Institutions of Higher Management, the American Council on 
Education and the Western Association of Schools and Colleges. 

Obviously, the accreditation process is not applicable 
to all of the respondents to the survey. However, for those 
that it does apply, the accreditation process appears to be 
designed to ensure the quality of course and program 
offerings. 

(2) Adequate Institutional Support 

Another possible method for ensuring quality is to 
have adequate institutional support for the educational course 
or program. 

The survey questionnaire asked several generic 
questions on institutional support. Those questions were: 

@® Question 32: Is your institution supported by a library 


containing extensive DoD financial management related 
materials? 
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@ Question 35: Is your program supported by a computer 
center or laboratory? 


@ Question 37: Is your program well supported with respect 
to administrative staff, funding for guest speakers, etc.? 
Survey respondents had mixed responses to these 
question. For example, 50 percent of respondents reported 
that their program was not supported by a library with 
extensive DoD material. Similarly, seven of nineteen 
respondents answered that their program was not well supported 
with respect to administrative staff, funding for guest 
speakers, etc. (However, those providing detailed answers (4 
responses) focused on funding for guest speakers as the only 
deficiency in this area] Nevertheless, it seems clear that 
adequate support is a measure of quality. 

While it may seem obvious that developing and 
maintaining quality programs necessitates the involvement not 
only of the provider but also of the sponsor and clients 
(i.e., the students and the "recipients" of these students), 
the overall response rate indicates that this is not the case. 
If there is a concern for quality, the imperative to 
understand and meet the needs of the client is obvious. 

Furthermore, the divergence of responses suggest that 
more effort on the part of DoD needs to be expended toward the 
development of quality as3urance methods that ensure financial 
management education and training is of the highest quality 


and is relevant, timely and comprehensive. 
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C. AREAS FOR FURTHER RESEARCH 

In researching the quality assurance methods employed by 
providers of financial management and education, several areas 
of further research were noted. These include: 

® Can a model of quality assurance be developed and applied 
to financial management education and training programs? 

@® Can the various quality assurance methods cited by 
financial management education and training providers be 
employed by all such providers? 

@® Are certain quality methods used by financial education 
and training programs more effective than others; and if 
so why? 

One final area for further research is the effort to 
obtain a 100 percent survey response in order to develop a 
complete and comprehensive data base on general 


characteristics and quality methods employed by financial 


management education and training programs. 
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APPENDIX A: 
DoD FINANCIAL MANAGEMENT SCHOOLS PERSONNEL 
CONTACTED IN SURVEY DEVELOPMENT 


Mr. William Pease 
US Army Finance School 
Fort Benjamin Harrison, IN 


Mr. Paul Wagner 
US Army Management Engineering College 
Rock Island Arsenal, IL 


Mr. Wade Cliendinst 
US Army Audit Agency 
Alexandria, VA 


Mr. Jack Mc Murchy 
US Army Logistics Management College 
Fort Lee, VA 


Mr. Robinson 
3750TH Technical Training Group 
Sheppard Air Force Base, TX 


Dr. Richard Lestor 

Air University 

Professional Military Comptroller School 
Maxwell Air Force Base, AL 


Lieutenant Colonel Stephen Adams 

Air University 

Professional Military Comptroller School 
Maxwell Air Force Base, AL 


Commander Glenn Eberling 

Naval Postgraduate School 
Professional Comptroller Course 
Monterey, CA 


Ms. Nina Allen 
American University 
Washington, DC 


Dr. C. W. Hoofnagle 


Fleet Training Center 
Norfolk, VA 
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APPENDIX B: 
DEPARTMENT OF DEFENSE FINANCIAL MANAGEMENT EDUCATION 
AND TRAINING QUALITY ASSESSMENT SURVEY 


PURPOSE: This survey questionnaire is designed to determine how 
DoD financial management education and training programs assess the 
quality of their programs. Quality in the context of this survey 
means providing accurate, valid, comprehensive and up-to-date 
information to meet the needs of your customers or clients. 
Quality begins with the determination of the need for the program 
and continues through the monitoring of the performance of 
graduates. Quality consists of providing the correct instruction 
for the appropriate target population. 


The following series of questions is designed to allow you to 
provide information on quality assessment techniques employed by 
your organization. We encourage you to answer the questions 
completely and add additional information wherever appropriate. 
Please attach additional pages and materials as necessary to 
provide complete answers. 

ORGANIZATIONAL INFORMATION: 
Respondent’s Name: 
Position: 


Agency/Component Name: 


Address: 


Phone Number: 


(DSN/Autovon) 
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Please provide the title or name of the instructional program 
offered at your institution: 











GENERAL PROGRAM INFORMATION: 











a What is the target audience for your program? 
(CIVILIAN) (MILITARY) (BOTH) 

2. What are the approximate percentages of class attendees? 
CIVILIAN: % 
MILITARY: % 

3. What are the ranges of grades or ranks of your attendees? 
CIVILIAN: 

(GS or GM - to GS or GM - ; SES Level) 

MILITARY: 

4. How many separate courses in financial management are 
offered within your program? 

i What is the length of your program in days, weeks or 
months? 

6. How many times per year is it offered? 

7. What is the annual enrollment in your program? 

8. What is the average (mean) class size? 
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9. If your program provides course credit for college, how 
many credits are offered in the total program? 


10. Do students completing the program receive a degree or 
certificate? 


(YES) (NO) 


If yes, what is the title of this award? 


The rest of the questionnaire is organized into the following 
parts: (1) Program requirements and development (2) Instruction (3) 


Ongoing evaluation and (4) Program assessment. Some of the 
questions require a simple YES/NO response, other ask for short 
responses, helpful comments and recommendations. We ask you to 


carefully complete this survey so that we can develop a 
comprehensive data base of quality assessment measures employed in 
financial management education and training. 


PROGRAM REOULTREMENTS AND DEVELOPMENT : 


11. How do you determine need for new courses or instruction? 
12. How do you determine the need to update or modify the 


instruction you currently offer? 











13. Do you have one or more sponsors or clients that directly 
influence your program of instruction? 


(YES) (NO) 


If yes, who is/are the sponsor(s)? 


14. Is course development guided or directed by higher 
authority? 
(YES) (NO) 


If so, who? 


15. What types of guidance do you receive? 

16. How often do you receive such guidance? 

a ty 2 Do you have staff dedicated to program development? 
(YES) (NO) 


If yes, how many? 


INSTRUCTION: 
18. What are the number of and percentages of civilian and/or 
military instructors in your program? 
CIVILIAN: Number Percentage 
MILITARY: Number Percentage 
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19. What policies and criteria are used to assign or hire 
Military faculty/instructors for your program? Please 
attech any material or guidance that describes these 
policies or criteria to your survey response. 


20. What policies and criteria are used to appoint or hire 
civilian faculty/instructors for your program? Please 
attach any material or guidance that describes these 
policies and criteria to your survey response. 


21. Are professional qualifications standards required for 
faculty/instructors (e.g., degrees, prior experience, 
completion of an instructor training program) ? 


(YES) (NO) 


If yes, please describe. Attach written guidance or material 
that describes such qualifications to your survey response. 


22. Do you use classroom observation to evaluate instruction? 
(YES) (NO) 


If yes, who performs this observation? 


How often is it performed? 
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23. Are other methods used to evaluate classroom instruction? 
(YES) (NO) 
If yes, please describe: 

24. Are other methods used to evaluate instructors? 
(YES) (NO) 
If yes, please describe: 

254 Do you employ productivity measures to evaluate 
instructors? 
(YES) (NO) 
If so, please describe. Attach any materials that 
describe this process to your survey response. 

26 Is your institution accredited? 


(YES) (NO) 


If yes, by whom? 


How often is the accreditation review performed? 








27. 


28. 


29. 





Is your institution reviewed by other agencies? 
(YES) (NO) 


If yes, by whom and how often? 


What is the nature and extent of this review? Please 
attach any materials that describes this process to your 
survey response. 


What methods or modes of instruction are utilized in your 
program (e.g., lecture, discussion, simulation, 
independent study) ? 





30. 


Es Fa 


What percentage of instruction is delivered in each mode 
you identified? 


What types of course reading materials are used in your 
program (e,g., textbooks, published articles, DoD official 
documents, self-generated materials) ? 
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32. Is your program supported by a library containing 
extensive DoD financial management related materials? 
(YES) (NO) 

33: How would you like to improve library support for your 
program? 

34. Are there improvements in your library services planned? 
(YES) (NO) 
If yes, please explain these improvements and indicate 
when they will occur. 

35% Is your program supported by a computer center or 
laboratory? 
(YES) (NO) 
If no, what computer resources are employed at your 
institution? 

36. Does your instructional program include computer exercises 


or assignments? 
(YES) (NO) 


If yes, please describe how computers are utilized. 
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37. Is your program well supported with respect to 
administrative staff, funding for guest speakers, etc.? 


(YES) (NO) 


What improvements in support would be desirable? 


ONGOING EVALUATION: 


38. What methods are used to determine the need for your 
current program? 


39. What methods are used to determine the validity, accuracy, 
relevance and timeliness of your current program? 


40. What methods are employed to determine student competence 
during and upon program completion (e.g., passing 
standardized tests, written course work, observation of 
performance on the job)? 


65 


41. 


42. 


43. 


44, 


Do you have a formal method of student course evaluation? 
(YES) (NO) 


If yes, describe this process and provide the form used to 
gather student evaluations with your response. 


Are other methods used to permit students to evaluate your 
instructional program? 


(YES) (NO) 


If yes, describe these methods: 


Are your students required to pass any standardized DoD or 
national examinations (e.g., CPA, CMA)? 


(YES) (NO) 
If yes, please identify: 


What percentage of students entering your program 
successfully complete it? 


PROGRAM ASSESSMENT: 


4s. 


Do you track the performance of your graduates? 


(YES) (NO) 


If yes, how is this done? 








46. 


47. 


48. 





Are formal methods employed to solicit feedback from 
former students and clients about the utility of your 
program? 


(YES) (NO) 


If yes, please explain how this is done and provide 
examples of such feedback. 


In your opinion, what are the most important methods you 
employ to ensure the quality of your program? 


Is there any other information that you would offer on 
quality assessment at your institution that we did not 
request? Please provide comments as appropriate. 
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COMPTROLLER OF THE DEPARTMENT OF DEFENSE 


WASHINGTON, DC 20301-1100 





FEB 27 1992 


MEMORANDUM FOR RECIPIENT OF THE FINANCIAL MANAGEMENT QUALITY 
ASSESSMENT SURVEY 


SUBJECT: Financial Management Quality Assessment Survey 


Our education and training community has been busy the past 
year analyzing how financial management instruction is provided 
throughout the Department. As part of this effort I have asked 
that a survey of quality assessment practices be undertaken to 
provide a baseline from which we can decide how to direct future 
initiatives for improving financial management education and 
training. 


The survey attached is intended to gather information on 
how Financial management education and training institutions 
currently.assure that the instruction they provide is of high 
quality and is as up-to-date as possible. I ask that you answer 
the survey questionnaire carefully and completely, and return it 
by the deadline indicated. I can assure you that this 
information will be used in determining how financial management 
education and training will be delivered in the future. Your 
contribution to the process of strengthening financial 
Management instruction is appreciated. 


42.0% 
ee 
Sean O'Keefe 


Attachment 
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APPENDIX C: 
DoD FINANCIAL MANAGEMENT AGENCIES AND DEPARTMENTS 


RESPONDING TO THE SURVEY 


NAN A NT H : 


US Army Finance School 
Fort Benjamin Harrison, IN 46216-5640 


US Army Management Engineering College 
Rock Island Arsenal, IL 61229-7040 


Syracuse University 
Army Programs Office 
310 School of Management 
Syracuse, NY 13244-2130 


US Army Engineer Division 
Directorate of CE Training Management 
Huntsville, AL 35807-4301 


US Army Logistics Management College 
Fort Lee, VA 23801-6056 


* The Judge Advocate General’s School 
Contract Law Division 
Charlottesville, VA 22903-1781 


* Center for Army Leadership 
Fort Leavenworth, KS 66027 


R FINAN MANAGEMENT HOOL 


3750TH Technical Training Group 

Sheppard Air Force Base; TX 76311-5434 

Air Force Institute of Technology 
Wright-Patterson Air Force Base, OH 45433-6583 
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Air University 
Professional Military Comptroller School 
Maxwell Air Force Base, AL 36112-5712 


2 NT HOOLS: 


Naval Postgraduate School 
Department of Administrative Sciences 
Monterey, CA. 93943-5022 


American University 
Washington, DC 20016-8070 


Troy State University 
P. O. Box 1032 
Fort Meyer, VA 22211 


Navy Comptroller 

Program Management Office 
Naval Air Station Pensacola 
Building 625D 

Pensacola, FL 32508-5175 


Fleet Training Center 
Norfolk, VA 23511 


Civil Engineer Corps Officers School 
Port Hueneme, CA 93043-5002 


Naval School of Health Sciences 
Bethesda, MD 20814-5033 


Fleet Aviation Specialized Operational Training Group 
Norfolk, VA 23511 


Navy School of Manpower Management 
Norfolk, VA 23511 


Naval Military Personnel Command 


Navy MWR Recreation Training Unit 
Patuxent River, MD 20670 
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MARINE CORPS FINANCIAL MANAGEMENT SCHOOLS: 


Financial Management School 
Marine Corps Service Support Schools 
Camp Lejeune, NC 28542-5050 


NOTE: An asterisks preceding the name indicates that a 
response was received. However, the response was that the 
agency or department did not perform financial management 
education and training. 





APPENDIX D: 
DoD FINANCIAL MANAGEMENT AGENCIES AND DEPARTMENTS 


WHICH FAILED TO RESPOND TO THE SURVEY 


MANAGEMENT. HOOLS : 


US Army Audit Agency 
3101 Park Center Drive 
Alexandria, VA 22302-1596 


US Army National Guard Bureau 
Washington, DC 20310-2500 


Army Reserves Readiness Training Center 
Fort McCoy, WI 54656-5000 


Office of the Deputy Chief of Staff for Personnel 
The Pentagon 

Washington, DC 20310-0300 

Defense Systems Management College 


Business Management Department 
Fort Belvoir, VA 22060-5426 


NT HOOLS : 


3400TH Technical Training Group 
Keesler Air Force Base, MS 39534-5000 


Ls: 


Service School Command 
San Diego, CA 92133-3000 


Naval Technical Training Center 
Meridian, MS 39309-5200 











Department of the Navy 
Program Information Center 
The Pentagon 

Washington, DC 20350 


AGENCY L: 


DLA Finance Center Training Section 
DCPSO-PF Building 150 

P. O. Box 3990 

Columbus, OH 43213-5000 


Note: The DLA Finance Training Center and the Defense 
Systems Management College responded to the survey after the 
primary research was complete. The Army Audit Agency is in 
the process of completing the survey. Their responses are 
not included in the analysis presented in Chapter IV or the 


detailed responses provided in Appendix E. 
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APPENDIX E: 
DETAILED SURVEY RESPONSES 

This appendix presents the responses to the survey 
questions. Appendix B is the actual survey instrument that 
was sent to all financial management education and training 
providers. See Appendix C for the listing of these agencies 
and departments which responded to the survey. Appendix D 
list those providers who did not respond to the survey. 
Chapter IV of this thesis summarizes the findings resulting 
from the survey. 

This appendix is organized by question. Those questions 
that required a "YES" or "NO" response are indicated by an 
asterisk preceding the question number. 

Each question is identified and followed by a summary of 
the responses. Where appropriate, comments provided by survey 


respondents are included. 
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1. What is the target audience for your program? 


The response option for this question was civilian, 


military or both. 


RESPONSE PERCENTAGE 
crvinran | 


MILITARY 3 
ote Ian 


While the vast majority of respondents replied "BOTH", the 





majority of students are milita:/. For example, several 
agencies replying "BOTH" reported military percentages in the 
90 percentile range. Tae sole agency reporting only 
"CIVILIAN" was American University. American University 
offers a graduate level program for civilian employees of the 


Department of the Navy. 


2. What are the approximate percentages of class 
attendees? 


The responses to this question varied, but as noted above 
the majority of students are military. Based on the 19 
surveys received, approximately 80 % of all reported students 


are military members. 
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3. What are the ranges of grades or ranks of your 
attendees? 







GRADE /RANK 
| sproressronay | | ak 
a 
aE ee) Seen Cae Mae | ee 


Of the 19 responses received, nine provide financial 







management education for "professionals". Professional in 
this analysis was defined as GS/GM 7’s and above or military 
officers. Only one respondent provided services for junior 
enlisted or civilian personnel. The balance of providers 
offered financial management education across the full 


spectrum of civilian and military pay grades. 


4. How many separate courses in financial management are 
offered within your program? 
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The range of courses offered as reported by survey 
respondents was from one to 36. Detailed reading of the 
responses indicate that providers of financial management 
education and training have difficulty defining courses. This 
particular question was designed to identify the number of 
separate courses vice the number of programs offered. It is 
not clear from the available survey data that consistency in 
course definition was employed by the respondents. 

For example, based on the survey responses, the total 
number of courses reported was 163. Based on information 
previously collected this number was expected to be 181. 
Further analysis is required to understand the cause of this 
discrepancy. The most likely cause of the discrepancy is the 
difficulty agencies have in identifying or categorizing 


courses as "financial management". 


Ss What is the length of your program in days, weeks or 
months? 


LENGTH 
5 DAYS OR LESS 
ONE MONTH 


1 MONTH - 1 YEAR 
ONE YEAR OR MORE 


TOTAL RESPONSES 
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This question again points out the difficulty in the 
distinction between "courses" and "programs". While in some 
cases they may be the same, e. g., the course and program 
constitute the only offering at an agency or department, the 
Majority of survey respondents replied on the length of 


courses vice programs. 


6. How man ime er year is it offered 


ee er 
| mwovorve | 6 | to 
ei el ee ae ic eee 
| more THAN TEN | | 


FREQUENCY PERCENTAGE 





ym What is the annual enrollment in your program? 







| tess muansso | 2] at 
ee ae a ee le cr 
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The average (mean ) enrollment based on the seventeen 


responses was 431 students. The range of students was from 36 
to 1,670. The annual total student population reported by the 


responding agencies totalled 7,324. 


8. What is the average (mean) class size? 






_MORE THAN THIRTY 1 A SF 





The average (mean) class size as reported by respondents 


was 23.4. The range was from 9 to 60 students. 


9. If your program provides course credit for college, 


how many credits are offered in the total program? 


COLLEGE CREDIT GRANTED PERCENT 
MASTER DEGREE a ae ee 
CREDITS ONLY 


NO COLLEGE CREDIT 
TOTAL RESPONSES 
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Eight responses were received on this question. Of the 
eight agencies responding, five were degree granting at the 
Master level with graduate credits ranging from 42 to 85. 
Three other agencies offered college credits of 6 hours, 16 


hours, and from 3 to 16 hours respectively. 


#10. Do students completing the program receive a degree or 
certificate? 


RESPONSE PERCENTAGE 





_TOTAL RESPONSES _ ; a eee eee 


rf w. i h f thi ward? 


Of the 19 agencies responding to this question, ten issued 
certificates of completion, two issued certificates of 
training, one issued a diploma and five issued masters 
degrees. Two agencies do not provide a degree or certificate 
to graduates. The total exceeds 19 due to one multiple 


response. 


80 











11. 





How do you determine need for new courses or 


instruction? 


The following comments were provided: 


Coordination with the Defense Finance and Accounting 
Service; Coordination with the Comptroller of the Army or 
Assistant Secretary of the Army for Financial Management; 
Coordination with the Army’s Training and Doctrine 
Command; and comments from the field. 


Changes in the operating environment in DoD. 


Feedback from participants; initiatives by faculty and 
directors of programs; guidance from the Assistant 
Secretary of the Army for Financial Management. 


Assigned course proponents assess preliminary needs prior 
to an annual market survey which determines definitive 
needs. 


New course start-up requirements are detailed in Army 
regulation 350-3 and are in the updated Training and 
Doctrine Command’s Training Requirements Analysis System. 


Inputs by users and field activities. 


New course requirements are identified by the customers, 
usually through the use of Air Force form 19 - "Request to 
Establish New Continuing Education Short Course". 
Biannual curriculum reviews with DoD customers, and 
proceeding of functional boards can also identify needs. 


Observation of changes in governmental and academic 
practices and problems. Consultation with the program 
sponsor. 


Communication with the Navy’s contract officer and adjunct 
professors who are professionals in the field. 


New course and instructional needs are determined by on- 


going feedback from students, alumni, faculty and our 
educational service officers. 
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12. 





Needs survey of major claimants, the career planning board 
and participation in financial management professional 
associations. 


Fleet inputs. 


Student feedback. Also input from the sponsor of the 
course. 


Input from the claimant; review of changes to official 
directives; input from field activities; input from class 
graduates; input from DoD; review of civilian sector 
approaches. 


Fleet needs; taskings from higher authority; critiques 
from other courses. 


Program changes; student and claimant feedback. 


Validated demands from field organizations for skills 
training. 


The normal cycle for determination of a new course of 
instruction begins with a change to operation in the 
community. The school will get input from headquarters 
that a task or course needs instruction. Initially, we 
get input from post-graduation surveys which indicate that 
new instruction is needed. Frequently, we distribute 
field surveys for more inputs. 


H rmin he n to update or modify the 


instruction you currently offer? 


The following comments were provided: 


Changes in regulations or new directives; changes in 
computer programs related to financial management; lessons 
learned, e. g., Desert Storm; results of enlisted and 
officer development tests. 
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Changes in regulations, directives or operating 
environment. 


Both courses have a two-dimensional evaluation program, 
subjective student end-of-course and instructor or 


facilitator evaluations. Courses are periodically 
reviewed by the course proponent, course manager and 
training quality assurance representative. Proponents 


provide input on regulatory and policy changes. 


We conduct post-graduation surveys with graduates and 
their supervisors biannually to assess customer 
satisfaction. 


Surveys, proponent inputs, MACOM inputs. 


Input from field or functional managers; system changes; 
data provided from occupational surveys. 


There are many possibilities. The faculty is responsible 
for maintaining currency in their field, as the field 
changes, so does the course. Other sources are end-of- 
course critiques, post-course critiques, curriculum 
reviews and communication with practitioners and sponsors. 


Current issues in financial management plus guidance from 
senior military and DoD comptrollers. 


Mid-course and end-of-course student evaluations. 

In order to determine the need to update or modify the 
instruction currently offered, we test alternative 
delivery methods of instruction. 


Interaction with students, instructional staff and policy 
makers from respective areas of subject matter. 


Annual course reviews and periodic curriculum reviews 
conducted by fleet or type commander staff. 


Student feedback, major claimant input along with changing 
issues in financial management. 


End-of-course critiques and surveys of past graduates. 


Course review; changes in reference material; changes in 
fleet needs. 
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® Continuous review of instructors and student critiques. 


@® The main determining factor for modifying our instruction 
is a change in the regulations governing financial 
Management. Post-graduation feedback questionnaires and 
several forms of internal feedback also help to determine : 
when updates need to be made. 


#13. Do you have one or more sponsors or clients that 
directly influence your program of instruction? 


RESPONSE PERCENTAGE 





If v who i r h on 


The following sponsors were specifically mentioned by 


survey respondents: 


® Secretary of the Air Force for Financial Management 
@ Defense Finance and Accounting Service 

@ Chief of Naval Operations 

@ Navy Comptroller 

@ Chief of Naval Education and Training 

@ Navy Supply Systems Command 


Fiscal Division, Headquarters, Marine Corps 








@ CINCLANTFLT, COMNAVAIRLANT, CNO 
@ Navy Bureau of Personnel 

@ DoD Comptroller 

@® Navy Bureau of Medicine 

@® NAVFAC 

@® Defense Acquisition University 
@® NCD, NCB, CFMCP 

@ Navy Exchange Command 


@ Navy Food Service Systems Command 


#14. r vel i r_dire d higher 





If so, who? 


Soldier Support Center, CASCOM, TRADOC, ASA (FM), DFAS 
® Assistant Secretary for the Army for Financial Management 
@ Corp of Engineers Training Issues Committee 


@® Headquarters, Air Training Command 


OP-82/Director of Office of Budgets and Reports, NCB 
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15. 





NCD, NCB, NCF 

Chief of Naval Technical Training 
CNET and NAVFAC 

BUMED 

COMNAVAIRLANT 


Standards Branch, Marine Corps 


Wha of idanc ° receive 


The following comments were provided: 


Regulations plus directed common core training. 
Learning objectives. 


Feedback from collective leadership and _ periodic 
assessment. 


Proponents provide guidance on and approve the technical 
content. 


Regulations and policy guidance. 


Subjects of concern are the appropriate content for 
courses and the levels of learning required for each. 


General guidelines as to curriculum content. 
Subject matter expertise. 


Selected topic material or requirements for graduate skill 
levels. 


Course review, on-site evaluations; constant feedback is 
received from sponsors. 
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16. 


was 





Curriculum reviews, directed input, issuance of new 
directives. 


Formal instructions. 
Internal curriculum review. 
The Marine Corps uses the Instructional Systems 


Development process known as the Systems Approach to 
Training (SAT) as a guideline for all course development. 


How often do you receive such guidance? 


The typical response to this question was that guidance 


received whenever necessary. Fourteen of the respondents 


answered in this fashion. Other responses were annually (1), 


quarterly (1), and two to three times a year (1). 


RESPONSE 





If yes, how many? 


The answers to this question varied from one to fifty- 


eight. The average response was between 1 and 3. 
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18. What are the number of and percentages of civilian 


and/or milita instructors in your program? 


Of the sixteen agencies providing detailed information on 


this question, the following cumulative data was obtained: 


CIVILIAN: Number 42 Percentage 47.5 
MILITARY: Number 157 Percentage 52.5 


The range of civilian employees was from 0 to 66; for 
military employees the range was 0 to 33. The average (mean) 
number of civilian employees was 6.3 (one agency with 66 
civilian employees was excluded from this average). The 


average (mean) number of military employees was 13.1. 


19. What policies and criteria are used to assign or hire 


mili £ 1 in rs for you rogram? 


The following responses were provided: 


® The officer or enlisted finance assignment branches screen 
and submit records of the best available personnel to the 
commandant who makes the final decision. 


@® Subject matter expertise, previous schooling, previous 
assignments, and enlisted/officer fitness reports are all 
included as part of the review prior to actual assignment. 


@® Grade, educational, experience and qualifications. 
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20. 


Military candidates are screened by a five member faculty 
committee. 


A Master’s deqree is required. Broad knowledge of the 
comptroller field with broad background ir at least one 
functional area. Strong performance record. 


Must have graduate degree and are financial management 
specialists. 


Must have subspecialty code; and 0-4 and above. 


Manpower efficiency reviews cond::cted by Chief of Naval 
Education and Training. 


Recent academic background and field experience. 
NEC 9502. 
New instructors are assigned on the basis of their 


technical knowledge of the subject matter they will teach 
and their communication skills. 


What policies and criteria are used to appoint or hire 


ivilian faculty/instructors for your program? 


The following comments were ~rovided: 
All civil service positions within the training instructor 
field have prescribec minimum qualifications. 


College degree, subject matter expertise, ability to 
teach. 


Coordination/consultation with department chairs and 
academic directors. 


Instructors are recommended by their supervisors. 


Review of job series and job experience. Depending on 
grade, career field experience is generally required. 
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Final selection is based on educational requirements, 
subject matter knowledge, experience, interview, and 
potential to perform on the "platform". 


@® Grade, educational experience, qualifications and OPM 
guidance. 


@® Civilian candidates are screened by a five member faculty 
committee. 


@ Must have doctoral degree. For senior faculty, must also 
have scholarly research record. 


@® Resumes, references, interviews. 


@ Level of degree, professional field experience in finance. 


*21. Are professional qualifications standards required for 


acul n tors (e.g., degrees rior ri e 
completion of an instructor training program) ? 





RESPONSE PERCENTAGE 
ee ee 
TOTAL RESPONSES 


If 1 rib 









@® College degree. 
@ Same standards as for the University. 


@ Most instructors have a degree in their subject area 
(minimum of bachelor’s) and not less than five years 
experience in their subject area. Further, instructors 
must complete the Instructional Methods Course. : ° 





All military officers have degrees; civilians usually have 
degrees. The Faculty Development Course must be completed 
within 90 days of assignment. 


Associates degree or higher; completion of Technical 
Training Instructor Course. 


All faculty must have an appropriate Master’s degree and 
experience in the field taught or a closely related field. 
Faculty hired without significant prior teaching 
experience must complete Academic Instructor School. 


Master’s degree plus completion of Air Force Academic 
Instructor School. 


We prefer faculty with Phd’s and teaching experience. 
Each faculty member in the teaching discipline at the 
master’s degree level will hold the terminal degree in the 
teaching discipline or a related discipline. Outstanding 
professional experience plus a Master’s degree is 
preferred. 

Specific Navy enlisted classification codes apply to some 
billets which require fleet experienced personnel to be 
Getailed to the billet. 


Minimum of Bachelor, desired Master’s and five years of 
experience. 


Instructor Training School; subspecialty designation. 
Graduate of instructor training school. 
Prior experience in the field. 


All instructors complete a two week Instructor Training 
Course. 
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TOTAL RESPONSES 








a ee ae 
aa ey eee ie ee eee 


100.0 








If wh rf hi bservation? 
The following responses were provided: 
Department directors, division chiefs, course directors, 
commandant. 
Staff and faculty development representatives. 
Peers. 
Program directors. 
Classroom courses are evaluated by the course manager, 
quality assurance representative, proponent or lead 
instructor. 
School deans, department chairman, course directors. 
Instructor supervisors and/or flight commander. 
Department head. 
Commandant. 
Regional academic director. 
Educational specialists. 


Staff instructors. 
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@® Academic Standards department. 

@ Academic Program division. 

@® Qualified instructors. 

@® Training director and department head. 

@ Master course manager. 

@ Curriculum Standards Branch Officer, Instructional 


Operations Officer, Academic Officer, Academic Chief, 
Instructional Operations Chief. 


How often is it performed? 


The following responses were provided: 


@ Weekly - 5 responses 

® Quarterly - 2 responses 

@® Twice yearly - 4 responses 
@ Annually - 6 responses 


@ Unscheduled or as appropriate - 2 responses 


#23. Are other methods used to evaluate classroom 
instruction? 







ae SO aE 
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all 





Of the fifteen agencies which responded to this question, 


cited student critiques as a method used to evaluate 


classroom instruction. Detailed responses are listed below: 


Student evaluation of training. 

End-of-class evaluation by students. 

Surveys from participants; group assessment sessions. 
End-of-course questionnaires by students;audits of classes 
by proponent representatives; post-graduation surveys sent 
to graduates and their supervisors 6 months after course 
completion; academic excellence analysis. 

ATC form 736; student critiques; Training evaluation 
Reports; Training Quality Reports from graduates and 
supervisors of graduates; field visits; STAN evaluations 
and Inspector General inspections. 

We use student/supervisor post course critiques. The 
student class leader debriefs the department head at the 
conclusion of each course. 

Student opinion questionnaires. 

Mid-course and end-of-course student evaluations. 


Student feedback instruments; discussions with 
instructors/administrators. 


Student: feedback from every class. Representatives from 
sponsor also monitor classes. 


Student critiques; peer review. 


Course reviews; student critiques; external evaluations 
(surveys) . : 


Quarterly improvement form completed by all students. 
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® Course evaluation upon student course completion and a 
supervisor’s evaluation of student performance in training 
skills 3 months after course completion. 


@ Every instructor completes an after instruction report to 
note any problems with lesson materials. Each student 
fills out an end-of-course critique prior to graduation 
which enables him to evaluate the overall course material. 


#24. Are other methods used to evaluate instructors? 







a ea a eee ee eee 
saat” ehaes Seis mos 36.8 






i 1 ribe 
The following responses were received: 


@® Selection of instructor of the month, quarter and year. 

@® Survey of a sample of students and their supervisors 
conducted six months after course completion to determine 
adequacy and application made of training. 


@ End-of-course questionnaires; performance appraisals; 
informal visits to classroom by course directors. 


@® Student critiques; STAN evaluations; Inspector General 
inspections. 


@® The student course critique covers the instructor’s 
performance. 
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@® Student opinion questionnaires. 

@® Faculty may choose a 15 minute tape presentation, arrange 
a supervisory visit and evaluation or submit a 
representative portfolio for evaluation purposes. 


® Student feedback instruments. Follow-up discussions with 
students/administrators. 


® Student critiques; peer review. 

@® Course review; student critiques; external evaluations. 

® Review of instructor prepared course material. 

@ 100 percent of the students of each class complete an 
Instructor Rating form. This form allows them to evaluate 


the lesson material as well as the effectiveness of the 
instructor. 


#25. 1 iv r val 





ae nee Pee eee ee 









rf 1 rib 


Only three of the nineteen respondents indicated that 
productivity measures were employed. Of these three only one 


respondent provided measures employed. These were: 
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instructor contact hours; student training years per 
instructor; instructor contact hours per student training 


years produced; and consulting hours. 





*26. I insti ion ited 


RESPONSE PERCENTAGE 
YES eee, Caen Peper: 1 eat 





If yes, by whom? 
@ North Central Association of Colleges and Universities, 
Commission on Institutions of Higher Education. 
@® AACSB for Schools of Management. 


@® Southern Association of Colleges and Schools (2 
responses). 


® North Central Association. 
@ American Council on Education (2 responses). 
@® Western Association of Schools and Colleges. 


@® Middle States Association. 


How often? 


Frequency varied from annually to every ten years. 


Typically, the review cycle is five years. 








If yes, by whom and how often? 


Responses received included: 


Army Training and Doctrine Command. 

American Council on Education (2 responses). 

Defense Finance and Accounting Agency; Secretary of the 
Air Force for Financial Management; Air Force Audit 
Agency; Community College of the Air Force. 

Air University (2 responses). 


National Association of Schools of Public Affairs and 
Administration. 


MIVER and QES. 

Commander, Training Command, U. S. Atlantic Fleet. 
CNET; CNTECTRA; NAVFAC. 

Navy and Bureau of Medicine Inspector’s General. 
Type commander. 


Command inspections. 


98 





28. 


What is the nature and extent of this review? 


The following responses were provided: 


Standard ACE review of course material and lesson plans. 
Courses and library support are evaluated by ACE teams of 
subject matter experts. Evaluators consider factors such 
as course objectives, subject matter, level of difficulty, 
duration, methods of student evaluation, applicability to 
post-secondary education programs, and background and 
selection of students and instructors. ACE then makes 
recommendations to colleges/universities for granting 
academic credit. 

To determine accuracy and currency of training. 
Primarily focuses on management. 

Curriculum review. 

Year-long self-assessment followed by site visit. 

Formal Inspector General inspections. 

Review of curriculum and adherence to directives. 


Inspections. 
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29. What methods or modes of instruction are utilized in 


ur rogram (e.g., lecture, discussion imulation 


independent study)? 





The typical responses to this question included lectures, 
discussions, case studies, simulation and self or independent 
study. All respondents save one cited lectures as the 


predominant mode of instruction. 
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30. What percentage of instruction is delivered in each 


mode you identified? 


Of the eleven institutions providing detailed percentages 
on modes of instruction, all cited lectures and discussions as 
the predominant modes utilized. On average, lectures 
represented 40 percent of the instructional program; 
discussion represented 20 percent. Simulation, case study and 
independent study represent approximately 10 percent each. 
The remaining 10 percent was divided among the modes reported 


in Question 29 above. 


31. What types of course reading materials are used in 
your. program (e,g., textbooks, published articles, 


DoD official ocumen self-generated materials)? 


Sixteen agencies responded to this question. Thirteen 


cited DoD official documents. Ten developed their own (self- 


generated) materials. And nine utilized textbooks. 
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#32. Is your program supported a libra containin 


extensive DoD financial management related 


materials? 












PERCENTAGE 
YES a eee eee eee 
TOTAL RESPONSES 100.0 





Of the eighteen responses received, half noted that they 
were not adequately supported by a library containing 
extensive DoD financially related materials. Question 33 and 
34 below ask specific question on library support. Six 
providers specified improvements that would like to see in 
library support (See Question 33) and six noted that 
improvements were planned in library support (See Question 
34). The issue not quantified in these series of questions is 
the importance or lack cf importance to the program of 


adequate library support. 
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33. How would you like to improve library support for your 
program? 


Six responses were provided on this question. They 


included: 


@ Establish a library within our school. 


@ Be able to interact with some DoD instructional material 
data base. 


@ Ensure that the library has all required materials. 


@ Improve library support by joining the Washington area 
consortium. 


@ Specified funds for financial texts. 


® Obtain sufficient copies of reference materials. 


*34. ar here rovement in our libra services 
planned? 












a ee ee ees ee ee 
Ss 
100.0 
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If 1 e lain these rovements and 


indicate when they will occur. 


@® An effort is underway to automate the Logistics Library 
which will provide computer assisted in-house services 
plus remote dial-in access. (Scheduled for FY 1993) 


@® Our library has just assigned collection development 
specialists to enhance library support for school 


programs. 
#35. I Yr progr. orted ac uter center or 
laboratory? 







eNO ee es ay 









If no, what computer resources are employed at your 
institution. 


Only one response was provided in response to this portion 
of the question; that response indicated that only personal 


computers were used. 











*36. Does your instructional program include computer 
exercises or assignments? 


RESPONSE PERCENTAGE 





If yes, please describe how computers are utilized. 


Responses included: 


@® Standard Army Information Management System. 

@® Real-world applications. 

@ Familiarize clients with computers. 

® Simulation. 

@® Data retrieval exercises; National Budget case. 
@® Self-paced instruction. 


® Application practices (4 responses). 
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*37. Is your program well supported with respect to 
administrative staff, funding for quest 


speakers,etc.? 





Of the four agencies that specified specific improvements, 
all noted additional funding for guest speakers would be 


desirable. 


38. meth ar rmine the need for your 


rr rogr: 


The following responses were provided: 

@ The Army conducts annual surveys of all users of Army 
training programs. 

@® Sponsors. 

@ Needs assessments are conducted by proponents prior to any 


consideration of course design or development. Proponents 
annually verify the need for classroom courses and an 
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annual market survey provided the needs verification from 
the users. 


The Total Army Centralized Individual Training 
Solicitation (TACITS) is a survey which is conducted 
annually for primary information and follow-up; Structure 
Manning Decision Review is conducted annually; the 
Training Resources Arbitration Panel (TRAP) process is 
done monthly; off-line memorandums can be submitted once 
the TRAP closes for execution; curriculum council reviews; 
annual course proponent reviews. 


Field evaluations; Training Evaluation reports; Training 
Review Analyses; occupational surveys. 


Curriculum review addresses this specific issue. The need 
is also assessed by the quantity of requests for any 
particular course. 


The need for resource management education among military 
services. 


Data on P-coded billets and other financial management 
positions. 


Program reviews are conducted every five years. 


Alumni surveys; student surveys; end-of-course 
assessments; ESO inputs; and faculty surveys. 


NC and NCD guidance; Career Program Planning Board; 
initiatives in the operational environment. 


Site surveys conducted by systems command sponsors and 
annual course curriculum reviews. 


Student feedback and sponsors’ requirements. 
Demand for student placements. 


Fleet need; tasking from higher authority; student 
critiques. 


Review by claimant. 


Extensive course review by program manager and field 
personnel. 
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@® The Marine Corps uses the Individual Training Standards 


39. 


that describe the specific tasks and knowledge required. 
We also conduct Course Content Review Boards to ensure the 
training standards are correct. 


meth r rmin he validi 
r relev. imelin of your current 
program? 


The following methods were cited: 


Annual review of training strategy by subject matter 
experts; anniial review of lesson plans; needs analysis. 


Proposed training program based on TACITS data is reviewed 
and approved by the Department of the Army. 


Accreditation and standards of graduate program. 


Proponents are charged with the responsibility for 
technical content. Proponent/instructor meetings as well 
as monitoring of courses are performed to ensure technical 
accuracy, timeliness, etc. Student end-of-course 
critiques are constantly monitored for student perceived 
requirements for change. 


Curriculum council reviews; annual course proponent 
reviews; end-of-course questionnaires; Structure Manning 
Decision review; Army Training Resource Requirements 
System. 


Field evaluations; Training Evaluation Reports; Training 
Review Analysis; occupational surveys; Inspector General 
inspections; student critiques. Also a customer service 
information line has been installed. 


Curriculum reviews; visits with the sponsor; student 
critiques; functional boards; and other feedback. 
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Review by senior financial management officials; student 
feedback. 


Observation of changes in governmental and academic 
practices and problems. Consultation with program 
sponsor. 


Surveys and interviews with students, alumni and 
practitioners in the field. 


All programs are under continuous review. Inputs from the 
teaching site is gathered each term to determine the 
validity of our current program. 


Subject matter experts; career board annual report, 
classroom visits, surveys of the community. 


Annual course reviews; systems command assessments and 
technical audits. 


Sponsor review; willingness of users to reimburse for the 
course. 


Curriculum survey to past graduates; review by sponsors. 
Course reviews; standardization conferences. 
Student/claimant feedback. 


Competent job performers and subject matter experts review 
course content for validity, accuracy and relevance. 


Following completion of each course, a career content 


review board is conducted to ensure the material is 
accurate, timely and relevant. 
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40. What methods are employed to determine student 


ompe ce during and upon program completion (e. 


passing standardized tests, written course work, 


ervation of rformance on the job)? 


Responses provided included: 


@ Practical exercises (4 responses) 

@ Tests (14 responses) 

@ Research papers (4 responses) 

@® Individual or group projects (3 responses) 
@® Case studies (4 responses) 


@® Oral presentations (3 responses) 





@ Passing national exams (2 responses) 


@® Thesis (1 response) 


*41. D u Vv af 1 method £ stu t ourse 


evaluation? 


RESPONSE PERCENTAGE 
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Of the nineteen responses received to this question, all 
indicated that a formal method of student course evaluation 


existed. All used a student critique form for this purpose. 


*42. Ar her meth d rmi udents to 


val rin ional program? 





RESPONSE PERCENTAGE 
TOTAL RESPONSES 









ri h. 


The following comments were provided: 

@® The Commandant holds periodic informal discussions with 
students. 

@® Post-graduation surveys (3 responses). 


@® DD form 1556 is completed by students and their 
supervisors. 


@® Student-to-instructor feedback. 
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@® Students provide informal comments to their class leaders, 
who can relay them at end-of-course debriefs. 


® Students are invited to meet with the department chairman 
periodically and can make individual comments to him as 
they wish. 


@ Follow-up with educational specialists. 


#43. Ar our students r ired to pa standardized 
DeD or national examinations (e.g., CPA, CMA)? 






RESPONSE 

YES see ee ee ee 
as eel eee ee 
ae ae ae eee 


The one respondent answering "yes" to this indicated 







_TOTAL RESPONSES _ 


that students are required to pass the GRE, GMAT or MAT. 


44. ercenta of students rin our program 
successfully complete it? 


Of the eleven responses to this question, 10 indicated 


that completion rates averaged well over 95 percent. One 


agency cited a completion rate of 75 percent. 
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#45. Do you track the performance of your graduates? 


| RESPONSE PERCENTAGE 





If yes, how is this done? 


Of the seven responses provided, six agencies cited the 
use Of post-graduate surveys as the principle method utilized 


to track graduate performance. Specific comments were: 


® School liaison teams visit finance units in the field to 
evaluate school training and training needs with graduates 
and their supervisors. Survey of a sample of students and 
their supervisors is conducted about six months after 
course completion to determine adequacy and applicability 
of training. 


® Post graduate surveys are sent to graduates and their 
supervisors six months after course completion. 


@ Field visits by training evaluators. 
@® We use one year and five year out surveys of graduates. 
@ Follow-up questionnaire one year after completion; survey 


of supervisors of graduates of our program; track 
progression through the alumni association. 


@® A post-course evaluation is sent to the student's 
supervisor. It is intended to determine the effectiveness 
of training more than the performance of the individual. 











@® The supervisors of all graduates are sent a feedback 
questionnaire to help us evaluate if the student learned 
the necessary prerequisite knowledge. 


#46. Are formal methods employed to solicit feedback from 
former u and clients about the utility of your 






ES 









Of the twelve responses received, nine agencies cited the 
use of post-graduate surveys for this purpose. Post-graduate 
surveys were used for two purposes: (1) To track the 
performance of graduates (see Question 45 above), and (2) To 
assess the effectiveness or utility of the program. 

Post-graduate surveys are extensively discussed in Chapter 


Iv. 
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47. 


the 


In your opinion, what are the most important methods 


you employ to ensure the quality of your program? 


This question was designed to be the cap-stone question of 


survey. It summarizes the methods used by survey 


respondents to ensure quality. Responses included: 


Visits to finance units in the field; selection of 
instructors/subject matter experts who have field 
experience; insuring that we are kept informed on the 
numerous changes in finance, accounting and resource 
management regulations and policies. 


Qualified faculty; feedback from students and their 
supervisors. 


Recognition by sponsors and commands/agencies which 
provide participants for the program; competition for 
individuals to attend; long standing reputation of the 
program. 


Needs assessment; task analysis; analysis of pre/post test 
results; quality assurance; proponent and course manager 
evaluations; periodic proponent/instructor/SME workshops 
for technical accuracy and relevance; analysis of student 
end-of-course evaluations. 


Close attention to post-graduation surveys; Close 
attention to comments on end-of-course questionnaires; 
continuing dialogue with students and others in the 
functional area; exceptional instruction; extensive 
knowledge of subject matter and regulatory requirements. 


Field visits; student evaluations; rotation of military 
instructors; field feedback; workshops to review training. 


We attempt to hire and maintain a top quality faculty. 


The Faculty Academics Standards Committee reviews each 
course every three years. 
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@® Constant review of the curriculum by the faculty plus 
biannual review by Air University and senior DoD 
comptrollers; student feedback. 


® Faculty involvement with the Navy and Defense Department 
and in academic activities and organizations. 


@ We set high standards for professional graduate level work 
and expect the faculty to meet these standards. 


@® Student course assessments; faculty observations. 


@® Review of material for accuracy of content; student and 
instructor evaluations; post-training job performance. 


@ Periodic fleet/type commander staff reviews and systems 
command technical assessments. 


@® Constant feedback from the students and inputs from the 
course sponsor. 


Collection of data via student critiques; trend analysis 
over time. 


® Fleet inputs. 


@ On-going student and client feedback. 


Feedback from students and their supervisors. 


® The Instructional Systems Development Process guides us to 
ensure the quality of our program. Additionally, our 
school enjoys an outstanding relationship with field 
personnel. We are able to respond to changes to 
regulations and provide better instruction. 


48. Is there any other information that you would offer on 


quality assessment at your institution that we did not 
request? 


No responses were provided to this question. 
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